{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1803c39-71f2-4564-bd69-3df66a1cb947",
   "metadata": {},
   "source": [
    "# Inspect Rich Documents with Gemini Multimodality and Multimodal RAG Challenge Lab\n",
    "\n",
    "__Note__: if you encounter an authentication error when running the cells in the notebook, go to __Vertex AI__ > __Dashboard__, and click on __Enable All Recommended APIs__. Then, re-run the failed cell, and continue the lab.  \n",
    "\n",
    "## Setup and requirements\n",
    "\n",
    "### Install Vertex AI SDK for Python and other dependencies\n",
    "\n",
    "Run the following four cells below before you get to Task 1. Be sure to add your current project ID to the cell titled __Define Google Cloud project information__. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "219e959e-a819-4af5-8ad2-db5cf77f969e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# Install required python packages and other dependencies\n",
    "!pip3 install --upgrade --user google-cloud-aiplatform\n",
    "\n",
    "!pip3 install --upgrade --user google-cloud-aiplatform pymupdf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "245e8b96-1d74-4fee-903d-66e10ec82d46",
   "metadata": {},
   "source": [
    "### Restart current runtime\n",
    "\n",
    "You must restart the runtime in order to use the newly installed packages in this Jupyter runtime. You can do this by running the cell below, which will restart the current kernel.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "864cef62-bfea-4843-b2a3-0b19bc00a0df",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "import IPython\n",
    "\n",
    "# Restart the kernet after libraries are loaded.\n",
    "\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92ea66ce-c9a5-478b-b95d-7900488d1960",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84e078e2-2891-42ed-95f7-ce31795045d0",
   "metadata": {},
   "source": [
    "### Define Google Cloud project information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d81ca78d-1320-49dd-8c43-3d5608f47c61",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "import sys\n",
    "\n",
    "# Define project information and update the location if it differs from the one specified in the lab instructions.\n",
    "PROJECT_ID = \"\"  # @param {type:\"string\"}\n",
    "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
    "\n",
    "# Try to get the PROJECT_ID automatically.\n",
    "if \"google.colab\" not in sys.modules:\n",
    "    import subprocess\n",
    "\n",
    "    PROJECT_ID = subprocess.check_output(\n",
    "        [\"gcloud\", \"config\", \"get-value\", \"project\"], text=True\n",
    "    ).strip()\n",
    "\n",
    "print(f\"Your project ID is: {PROJECT_ID}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48216434",
   "metadata": {},
   "source": [
    "### Initialize Vertex AI\n",
    "\n",
    "Initialize the Vertex AI SDK for Python for your project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f5ded7d-13b9-42ff-87dc-0b5c998f884f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# Initialize Vertex AI.\n",
    "import vertexai\n",
    "\n",
    "vertexai.init(project=PROJECT_ID, location=LOCATION)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb712712-c7fb-43ef-be19-2497d610111c",
   "metadata": {},
   "source": [
    "## Task 1. Generating Multimodal Insights with the Gemini Pro Vision model\n",
    "\n",
    "Gemini 1.0 Pro Vision (gemini-1.0-pro-vision) is a multimodal model that supports multimodal prompts. You can include text, image(s), and video in your prompt requests and get text or code responses.\n",
    "\n",
    "To complete Task 1, follow the instructions at the top of each notebook cell:\n",
    "* Run the cells with the comment \"RUN THIS CELL AS IS\".\n",
    "* Complete and run the cells with the comment \"COMPLETE THE MISSING PART AND RUN THIS CELL\".\n",
    "\n",
    "__Note__: Ensure you can see the weather related data in the response that is printed.\n",
    "\n",
    "\n",
    "### Setup and requirements for Task 1\n",
    "\n",
    "#### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85d32222-695d-4993-bdc4-fde7bc7948f1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "from vertexai.generative_models import (\n",
    "    GenerationConfig,\n",
    "    GenerativeModel,\n",
    "    Image,\n",
    "    Part,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "435fad59-d08b-4aef-8230-7f567643ce14",
   "metadata": {},
   "source": [
    "#### Load Gemini 1.0 Pro Vision model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd63f204-6acc-41aa-ac25-45544d5c3888",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "multimodal_model = GenerativeModel(\"gemini-1.0-pro-vision\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0c3a051-d567-4146-b428-fcb285c6485d",
   "metadata": {},
   "source": [
    "#### Define helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8de36ca-ded6-46e4-99e9-4ed2fc88ee8b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "import http.client\n",
    "import typing\n",
    "import urllib.request\n",
    "\n",
    "import IPython.display\n",
    "from PIL import Image as PIL_Image\n",
    "from PIL import ImageOps as PIL_ImageOps\n",
    "\n",
    "\n",
    "def display_images(\n",
    "    images: typing.Iterable[Image],\n",
    "    max_width: int = 600,\n",
    "    max_height: int = 350,\n",
    ") -> None:\n",
    "    for image in images:\n",
    "        pil_image = typing.cast(PIL_Image.Image, image._pil_image)\n",
    "        if pil_image.mode != \"RGB\":\n",
    "            # RGB is supported by all Jupyter environments (e.g. RGBA is not yet)\n",
    "            pil_image = pil_image.convert(\"RGB\")\n",
    "        image_width, image_height = pil_image.size\n",
    "        if max_width < image_width or max_height < image_height:\n",
    "            # Resize to display a smaller notebook image\n",
    "            pil_image = PIL_ImageOps.contain(pil_image, (max_width, max_height))\n",
    "        IPython.display.display(pil_image)\n",
    "\n",
    "\n",
    "def get_image_bytes_from_url(image_url: str) -> bytes:\n",
    "    with urllib.request.urlopen(image_url) as response:\n",
    "        response = typing.cast(http.client.HTTPResponse, response)\n",
    "        image_bytes = response.read()\n",
    "    return image_bytes\n",
    "\n",
    "\n",
    "def load_image_from_url(image_url: str) -> Image:\n",
    "    image_bytes = get_image_bytes_from_url(image_url)\n",
    "    return Image.from_bytes(image_bytes)\n",
    "\n",
    "\n",
    "def display_content_as_image(content: str | Image | Part) -> bool:\n",
    "    if not isinstance(content, Image):\n",
    "        return False\n",
    "    display_images([content])\n",
    "    return True\n",
    "\n",
    "\n",
    "def display_content_as_video(content: str | Image | Part) -> bool:\n",
    "    if not isinstance(content, Part):\n",
    "        return False\n",
    "    part = typing.cast(Part, content)\n",
    "    file_path = part.file_data.file_uri.removeprefix(\"gs://\")\n",
    "    video_url = f\"https://storage.googleapis.com/{file_path}\"\n",
    "    IPython.display.display(IPython.display.Video(video_url, width=600))\n",
    "    return True\n",
    "\n",
    "\n",
    "def print_multimodal_prompt(contents: list[str | Image | Part]):\n",
    "    \"\"\"\n",
    "    Given contents that would be sent to Gemini,\n",
    "    output the full multimodal prompt for ease of readability.\n",
    "    \"\"\"\n",
    "    for content in contents:\n",
    "        if display_content_as_image(content):\n",
    "            continue\n",
    "        if display_content_as_video(content):\n",
    "            continue\n",
    "        print(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea68bb17-db90-437a-89b1-d1fe1d2e715f",
   "metadata": {},
   "source": [
    "### Task 1.1. Image understanding across multiple images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bcae938-8299-4641-87d9-d78c3dc88ac5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# You're going to work with provided variables in this task. \n",
    "# First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "image_ask_first_1_url = \"https://storage.googleapis.com/spls/gsp520/Google_Branding/Ask_first_1.png\"\n",
    "image_dont_do_this_1_url = \"https://storage.googleapis.com/spls/gsp520/Google_Branding/Dont_do_this_1.png\"\n",
    "image_ask_first_1 = load_image_from_url(image_ask_first_1_url)\n",
    "image_dont_do_this_1 = load_image_from_url(image_dont_do_this_1_url)\n",
    "\n",
    "instructions = \"Instructions: Consider the following image that contains text:\"\n",
    "prompt1 = \"What is the title of this image\"\n",
    "prompt2 = \"\"\"\n",
    "Answer the question through these steps:\n",
    "Step 1: Identify the title of each image by using the filename of each image.\n",
    "Step 2: Describe the image.\n",
    "Step 3: For each image, describe the actions that a user is expected to take.\n",
    "Step 4: Extract the text from each image as a full sentence.\n",
    "Step 5: Describe the sentiment for each image with an explanation.\n",
    "\n",
    "Answer and describe the steps taken:\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "867dc55c",
   "metadata": {},
   "source": [
    "#### Create an input for the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e390c4b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Now, you're going to create an input for your multimodal model. Create your contents list using the variables above. Ensure the structure matches the format expected by the multimodal model.\n",
    "\n",
    "contents = \"ENTER YOUR CODE HERE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac4aad87",
   "metadata": {},
   "source": [
    "#### Generate responses from the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7156c265",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the next part of this task, you're going to generate responses from the multimodal model. Capture the output of the model in the \"responses\" variable by using your \"contents\" list.\n",
    "\n",
    "\n",
    "responses = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a28025b0",
   "metadata": {},
   "source": [
    "#### Display the prompt and responses\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc92127e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the last part of this task, you're going to print your contents and responses with the prompt and responses title provided. Use descriptive titles to help organize the output (e.g., \"Prompts\", \"Model Responses\") and then display the prompt and responses by using the print() function. \n",
    "\n",
    "# Hint: \"\\n\" inserts a newline character for clearer separation between the sections.\n",
    "\n",
    "\n",
    "\"ENTER YOUR CODE HERE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0527c725-e400-4838-991b-33cd0772e450",
   "metadata": {
    "tags": []
   },
   "source": [
    "### To verify your work for Task 1.1, click __Check my progress__ in the lab instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa5b2269-4982-4c77-85b7-b6413b17f293",
   "metadata": {},
   "source": [
    "### Task 1.2. Similarity/Differences between images\n",
    "\n",
    "#### Explore the variables of the task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "323aa40f-18dd-4454-a9f1-474d4c4c721b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# You're going to work with provided variables in this task. First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "\n",
    "image_ask_first_3_url = \"https://storage.googleapis.com/spls/gsp520/Google_Branding/Ask_first_3.png\"\n",
    "image_dont_do_this_3_url =  \"https://storage.googleapis.com/spls/gsp520/Google_Branding/Dont_do_this_3.png\"\n",
    "image_ask_first_3 = load_image_from_url(image_ask_first_3_url)\n",
    "image_dont_do_this_3 = load_image_from_url(image_dont_do_this_3_url)\n",
    "\n",
    "prompt1 = \"\"\"\n",
    "Consider the following two images:\n",
    "Image 1:\n",
    "\"\"\"\n",
    "prompt2 = \"\"\"\n",
    "Image 2:\n",
    "\"\"\"\n",
    "prompt3 = \"\"\"\n",
    "1. What is shown in Image 1 and Image 2?\n",
    "2. What is similar between the two images?\n",
    "3. What is difference between Image 1 and Image 2 in terms of the text ?\n",
    "\"\"\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5bb6c29",
   "metadata": {},
   "source": [
    "#### Create an input for the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "420a9d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Now, you're going to create an input for your multimodal model. Create your contents list using the variables above. Ensure the structure matches the format expected by the multimodal model.\n",
    "\n",
    "contents = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68eaa88b",
   "metadata": {},
   "source": [
    "#### Set configuration parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bdc9ef8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Now, you're going to set configuration parameters that will influence how the multimodal model generates text. These settings control aspects like the creativity and focus of the responses. Here's how:\n",
    "# Temperature: Controls randomness. Lower values mean more predictable results, higher values mean more surprising and creative outpu\n",
    "# Top p / Top k: Affects how the model chooses words. Explore different values to see how they change the results.\n",
    "# Other parameters: Check the model's documentation for additional options you might want to adjust.\n",
    "\n",
    "# Store your configuration parameters in a generation_config variable. This improves reusability, allowing you to easily apply the same settings across tasks and make adjustments as needed.\n",
    "\n",
    "generation_config = \"ENTER YOUR CODE HERE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a78eecbb",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Generate responses from the multimodal model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74696b35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the next part of this task, you're going to generate responses from a multimodal model. capture the output of the model in the \"responses\" variable by using your \"contents\" list and the confiuguration settings.\n",
    "\n",
    "\n",
    "responses = \"ENTER YOUR CODE HERE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16501a73",
   "metadata": {},
   "source": [
    "#### Display the prompt and responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28236144",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the last part of this task, you're going to print your contents and responses with the prompt and responses title provided. Use descriptive titles to help organize the output (e.g., \"Prompts\", \"Model Responses\") and then display the prompt and responses by using the print() function. \n",
    "\n",
    "# Hint: \"\\n\" inserts a newline character for clearer separation between the sections.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8ab2ba5-7789-43ff-aaea-88cfcb75a4f9",
   "metadata": {},
   "source": [
    "### To verify your work for Task 1.2, click __Check my progress__ in the lab instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f815778c-aee0-4c3d-9d98-445ec693b18a",
   "metadata": {},
   "source": [
    "### Task 1.3. Generate a video description\n",
    "\n",
    "#### Explore the variables of the task\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90071041-d4c0-4bf2-a783-853dadaa3dca",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# You're going to work with provided variables in this task. \n",
    "# First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "\n",
    "prompt = \"\"\"\n",
    "What is the product shown in this video?\n",
    "What specific product features are highlighted in the video?\n",
    "Where was this video filmed? Which cities around the world could potentially serve as the background in the video?\n",
    "What is the sentiment of the video?\n",
    "\"\"\"\n",
    "video = Part.from_uri(\n",
    "    uri=\"gs://spls/gsp520/google-pixel-8-pro.mp4\",\n",
    "    mime_type=\"video/mp4\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f19c1b8b",
   "metadata": {},
   "source": [
    "#### Create an input for the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b12f7b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Now, you're going to create an input for your multimodal model. Create your contents list using the variables above. Ensure the structure matches the format expected by the multimodal model.\n",
    "\n",
    "contents = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9f4971b",
   "metadata": {},
   "source": [
    "#### Generate responses from the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17e1a6e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the next part of this task, you're going to generate responses from a multimodal model. Capture the output of the model in the \"responses\" variable by using your \"contents\" list.\n",
    "\n",
    "\n",
    "responses = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fa2d51f",
   "metadata": {},
   "source": [
    "#### Display the prompt and responses\n",
    "\n",
    "**Note:** If you encounter any authentication error below cell run, go to the **Navigation menu**, click **Vertex AI > Dashboard**, then click **\"Enable all Recommended APIs\"** Now, go back to cell 16, and run cells 16, 17 and below cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef32b772",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the last part of this task, you're going to print your contents and responses with the prompt and responses title provided. Use descriptive titles to help organize the output (e.g., \"Prompts\", \"Model Responses\") and then display the prompt and responses by using the print() function. \n",
    "\n",
    "# Hint: \"\\n\" inserts a newline character for clearer separation between the sections.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b4574b5-8ae4-4212-85d7-badc66e9d3df",
   "metadata": {},
   "source": [
    "Proceed to Task 1.4 below (no progress check for Task 1.3 in lab instructions). \n",
    "\n",
    "### Task 1.4. Extract tags of objects throughout the video\n",
    "\n",
    "#### Explore the variables of the task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69b54344-7190-46a4-877f-c9c9f7daa7a8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# You're going to work with provided variables in this task. First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "prompt = \"\"\"\n",
    "Answer the following questions using the video only:\n",
    "\n",
    "Which particular sport is highlighted in the video?\n",
    "What values or beliefs does the advertisement communicate about the brand?\n",
    "What emotions or feelings does the advertisement evoke in the audience?\n",
    "Which tags associated with objects featured throughout the video could be extracted?\n",
    "\"\"\"\n",
    "video = Part.from_uri(\n",
    "    uri=\"gs://spls/gsp520/google-pixel-8-pro.mp4\",\n",
    "    mime_type=\"INSERT CORRECT MIME TYPE\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2885e5bb",
   "metadata": {},
   "source": [
    "#### Create an input for the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab39bda2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Now, you're going to create an input for your multimodal model. Create your contents list using the variables above. Ensure the structure matches the format expected by the multimodal model.\n",
    "\n",
    "contents = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce38fe28",
   "metadata": {},
   "source": [
    "#### Generate responses from the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "442a83c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the next part of this task, you're going to generate responses from a multimodal model. capture the output of the model in the \"responses\" variable by using your \"contents\" list and video input.\n",
    "\n",
    "\n",
    "responses = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b090def3",
   "metadata": {},
   "source": [
    "#### Display the prompt and responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6cba353",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the last part of this task, you're going to print your contents and responses with the prompt and responses title provided. Use descriptive titles to help organize the output (e.g., \"Prompts\", \"Model Responses\") and then display the prompt and responses by using the print() function. \n",
    "\n",
    "# Hint: \"\\n\" inserts a newline character for clearer separation between the sections.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0932722c-fd8e-411a-a09e-ff1e530ba531",
   "metadata": {},
   "source": [
    "Proceed to Task 1.5 below (no progress check for Task 1.4 in lab instructions). \n",
    "\n",
    "### Task 1.5. Ask more questions about a video\n",
    "\n",
    "**Note:** Although this video contains audio, Gemini does not currently support audio input and will only answer based on the video.\n",
    "\n",
    "#### Explore the variables of the task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c7d9b2e-f167-473e-8fce-65d677ec5ff9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# You're going to work with provided variables in this task. \n",
    "# First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "\n",
    "prompt = \"\"\"\n",
    "Answer the following questions using the video only:\n",
    "\n",
    "How does the advertisement portray the use of technology, specifically AI, in capturing and preserving memories?\n",
    "What visual cues or storytelling elements contribute to the nostalgic atmosphere of the advertisement?\n",
    "How does the advertisement depict the role of friendship and social connections in enhancing experiences and creating memories?\n",
    "Are there any specific features or functionalities of the phone highlighted in the advertisement, besides its AI capabilities?\n",
    "\n",
    "Provide the answer JSON.\n",
    "\"\"\"\n",
    "video = INSERT CORRECT CLASS NAME.from_uri(\n",
    "    uri=\"gs://spls/gsp520/google-pixel-8-pro.mp4\",\n",
    "    mime_type=\"video/mp4\",\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae45d655",
   "metadata": {},
   "source": [
    "#### Create an input for the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37f7cb3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Now, you're going to create an input for your multimodal model. Create your contents list using the variables above. Ensure the structure matches the format expected by the multimodal model.\n",
    "\n",
    "contents = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c61307a2",
   "metadata": {},
   "source": [
    "#### Generate responses from the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dea2a83",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the next part of this task, you're going to generate responses from a multimodal model. capture the output of the model in the \"responses\" variable by using your \"contents\" list and video input.\n",
    "\n",
    "\n",
    "responses = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "987f77f7",
   "metadata": {},
   "source": [
    "#### Display the prompt and responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d993e0c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the last part of this task, you're going to print your contents and responses with the prompt and responses title provided. Use descriptive titles to help organize the output (e.g., \"Prompts\", \"Model Responses\") and then display the prompt and responses by using the print() function. \n",
    "\n",
    "# Hint: \"\\n\" inserts a newline character for clearer separation between the sections.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b238a1a4-7b7c-463f-af54-d853f1786b73",
   "metadata": {},
   "source": [
    "Proceed to Task 1.6 below (no progress check for Task 1.5 in lab instructions). \n",
    "\n",
    "### Task 1.6. Retrieve extra information beyond the video\n",
    "\n",
    "#### Explore the variables of the task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1668408-6c55-460f-b687-5916b417ed85",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# You're going to work with provided variables in this task. \n",
    "# First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "\n",
    "prompt = \"\"\"\n",
    "Answer the following questions using the video only:\n",
    "\n",
    "How does the advertisement appeal to its target audience through its messaging and imagery?\n",
    "What overall message or takeaway does the advertisement convey about the brand and its products?\n",
    "Are there any symbolic elements or motifs used throughout the advertisement to reinforce its central themes?\n",
    "What is the best hashtag for this video based on the description ?\n",
    "\n",
    "\"\"\"\n",
    "video = Part.from_uri(\n",
    "    uri=\"INSERT VIDEO URI FROM PREVIOUS TASK\",\n",
    "    mime_type=\"video/mp4\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868429f1",
   "metadata": {},
   "source": [
    "#### Create an input for the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ec4cd5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Now, you're going to create an input for your multimodal model. Create your contents list using the variables above. Ensure the structure matches the format expected by the multimodal model.\n",
    "\n",
    "contents = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b46261e",
   "metadata": {},
   "source": [
    "#### Generate responses from the multimodal model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65bb93f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the next part of this task, you're going to generate responses from a multimodal model. capture the output of the model in the \"responses\" variable by using your \"contents\" list and video input.\n",
    "\n",
    "\n",
    "responses = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa426629",
   "metadata": {},
   "source": [
    "#### Display the prompt and responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c492f1f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# In the last part of this task, you're going to print your contents and responses with the prompt and responses title provided. Use descriptive titles to help organize the output (e.g., \"Prompts\", \"Model Responses\") and then display the prompt and responses by using the print() function. \n",
    "\n",
    "# Hint: \"\\n\" inserts a newline character for clearer separation between the sections.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f908d1-b54f-4d8d-993e-faec763d7a37",
   "metadata": {},
   "source": [
    "### To verify your work for Task 1.6, click __Check my progress__ in the lab instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42950158-1e1f-40bd-97b4-d18474dd86fe",
   "metadata": {},
   "source": [
    "## Task 2. Retrieving and integrating knowledge with multimodal retrieval augmented generation (RAG)\n",
    "\n",
    "To complete Task 2, follow the instructions at the top of each notebook cell:\n",
    "* Run the cells with the comment \"RUN THIS CELL AS IS\".\n",
    "* Complete and run the cells with the comment \"COMPLETE THE MISSING PART AND RUN THIS CELL\".\n",
    "\n",
    "For additional information about the available data and helper functions for Task 2, review the section named __Available data and helper functions for Task 2__ in the lab instructions.\n",
    "\n",
    "### Setup and requirements for Task 2\n",
    "\n",
    "#### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c106e655-0da7-4eba-8595-706622ab7ef3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# Import necessary packages and libraries.\n",
    "\n",
    "from IPython.display import Markdown, display\n",
    "from vertexai.generative_models import (\n",
    "    Content,\n",
    "    GenerationConfig,\n",
    "    GenerationResponse,\n",
    "    GenerativeModel,\n",
    "    HarmCategory,\n",
    "    HarmBlockThreshold,\n",
    "    Image,\n",
    "    Part,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "860ce78c",
   "metadata": {},
   "source": [
    "#### Load the Gemini 1.0 Pro and Gemini 1.0 Pro Vision model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be19c04e-b7d2-4439-a0bd-58ffa0c1f2dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# Load the Gemini 1.0 Pro and Gemini 1.0 Pro Vision model.\n",
    "\n",
    "text_model = GenerativeModel(\"gemini-1.0-pro\")\n",
    "multimodal_model = GenerativeModel(\"gemini-1.0-pro-vision\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb962cea-0b31-43f6-b757-90bc1ff1cd5b",
   "metadata": {},
   "source": [
    "#### Download custom Python modules and utilities \n",
    "\n",
    "The cell below will download some helper functions needed for this notebook, to improve readability. You can also view the code (`intro_multimodal_rag_utils.py`) directly on [Github](https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/retrieval-augmented-generation/utils/intro_multimodal_rag_utils.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12aa75ac-3512-4ee2-a4b6-e8e926d19587",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# Import necessary packages and libraries.\n",
    "import os\n",
    "import urllib.request\n",
    "import sys\n",
    "\n",
    "if not os.path.exists(\"utils\"):\n",
    "    os.makedirs(\"utils\")\n",
    "\n",
    "\n",
    "# Download the helper scripts from utils folder.\n",
    "url_prefix = \"https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/retrieval-augmented-generation/utils/\"\n",
    "files = [\"intro_multimodal_rag_utils.py\"]\n",
    "\n",
    "for fname in files:\n",
    "    urllib.request.urlretrieve(f\"{url_prefix}/{fname}\", filename=f\"utils/{fname}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3a4c367-585d-4e15-883d-ed0a674475f1",
   "metadata": {},
   "source": [
    "#### Get documents and images from Cloud Storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b83f06b4-4e9c-449b-94f3-4175b37777c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# Download documents and images used in this notebook.\n",
    "\n",
    "!gcloud storage rsync --recursive gs://spls/gsp520 .\n",    "print(\"Download completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b72d16-d530-424b-934a-ebf091bc254c",
   "metadata": {},
   "source": [
    "### Task 2.1. Build metadata of documents containing text and images\n",
    "\n",
    "__Note__: These steps may take a few minutes to complete.\n",
    "\n",
    "#### Import helper functions to build metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "442e1759-0c37-44ff-9f2b-85ee427e7f9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# Import helper functions from utils.\n",
    "from utils.intro_multimodal_rag_utils import get_document_metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dcfd03c-9713-4621-b5e5-fee645d338e9",
   "metadata": {},
   "source": [
    "#### Explore the variables of the task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c2b6d28-4d64-4e03-b395-23bd2c8b6bbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# You're going to work with provided variables in this task. \n",
    "# First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "\n",
    "# Specify the \"PDF folder path\" with single PDF and \"PDF folder\" with multiple PDF.\n",
    "\n",
    "pdf_folder_path = \"Google_Branding/\"  # if running in Vertex AI Workbench.\n",
    "\n",
    "# Specify the image description prompt. Change it\n",
    "image_description_prompt = \"\"\"Explain what is going on in the image.\n",
    "If it's a table, extract all elements of the table.\n",
    "If it's a graph, explain the findings in the graph.\n",
    "Do not include any numbers that are not mentioned in the image.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60611e48",
   "metadata": {},
   "source": [
    "\n",
    "#### Extract and store metadata of text and images from a document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca9d2331",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Call the \"get_document_metadata\" function from the utils file to extract text and image metadata from the PDF document. Store the results in two different DataFrames: \"text_metadata_df\" and \"image_metadata_df\".  \n",
    "# text_metadata_df: This will contain extracted text snippets, their corresponding page numbers, and potentially other relevant information.\n",
    "# image_metadata_df: This will contain descriptions of the images found in the PDF (if any), along with their location within the document.\n",
    "\n",
    "text_metadata_df, image_metadata_df = \"ENTER YOUR CODE HERE\"\n",
    "\n",
    "print(\"\\n\\n --- Completed processing. ---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a96cd147-4ce1-4110-9c5b-d755e7a07457",
   "metadata": {},
   "source": [
    "#### Inspect the processed text metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f471508-1a38-4e3b-a4be-47e82acaf54b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Explore the text_metadata_df dataframe by displaying the first few rows of the dataframe.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ed0c6a-3d40-497b-b2c5-4a86c3b65c8d",
   "metadata": {},
   "source": [
    "#### Import the helper functions to implement RAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4580d12-94a1-4e1b-8ff6-587916c85241",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Import helper functions from utils.\n",
    "\n",
    "from INSERT CLASS NAME.intro_multimodal_rag_utils import (\n",
    "    get_similar_text_from_query,\n",
    "    print_text_to_text_citation,\n",
    "    get_similar_image_from_query,\n",
    "    print_text_to_image_citation,\n",
    "    get_gemini_response,\n",
    "    display_images,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f110046a-0615-4836-8776-113bba1b4088",
   "metadata": {},
   "source": [
    "Proceed to Task 2.2 below (no progress check for Task 2.1 in lab instructions). \n",
    "\n",
    "### Task 2.2. Create a user query\n",
    "\n",
    "#### Explore the variables of the task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ad2d8f5-f937-44fc-9c2d-2a1426134bb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# You're going to work with provided variables in this task. \n",
    "# First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "query = \"\"\"Questions:\n",
    " - What are the key expectations that users can have from Google regarding the provision and development of its services?\n",
    "- What specific rules and guidelines are established for users when using Google services?\n",
    "- How does Google handle intellectual property rights related to the content found within its services, including content owned by users, Google, and third parties? \n",
    "- What legal rights and remedies are available to users in case of problems or disagreements with Google?\n",
    "- How do the service-specific additional terms interact with these general Terms of Service, and which terms take precedence in case of any conflicts?\n",
    " \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7d1fa0d-3b6f-44ff-bcbf-8c8b184f9ba7",
   "metadata": {},
   "source": [
    "Proceed to Task 2.3 below (no progress check for Task 2.2 in lab instructions).\n",
    "\n",
    "### Task 2.3. Get all relevant text chunks\n",
    "\n",
    "#### Retrieve relevant chunks of text based on the query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfbc25d9-2d7e-433e-a5df-8d38eae9e75f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Call the \"get_similar_text_from_query\" function from the utils file to retrieve relevant chunks of text based on the query. Store the results in a dictionart called \"matching_results_chunks_data\".  \n",
    "# matching_results_chunks_data: This dictionary will contain file_name, page_num, cosine_score, chunk_number and chunk_socre. The dictionary represents a search result for a query related to the text_metadata_df.\n",
    "\n",
    "matching_results_chunks_data = \"ENTER YOUR CODE HERE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e4ec297-ca76-4cee-a008-c22002a0f7d3",
   "metadata": {},
   "source": [
    "#### Display the first item of the text chunk dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5d72676-9721-4ade-9342-5ffa4dec45ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Explore the first item in your matching_results_chunks_data dictionary by displaying the first item.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f86a088-41c6-4bd2-8fc7-45f0a7e8a2e4",
   "metadata": {},
   "source": [
    "Proceed to Task 2.4 below (no progress check for Task 2.3 in lab instructions).\n",
    "\n",
    "### Task 2.4. Create context_text\n",
    "\n",
    "#### Create a list to store the combined chunks of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1595d30",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "\n",
    "# Create an empty list named \"context_text\". This list will be used to store the combined chunks of text.\n",
    "context_text = \"ENTER YOUR CODE HERE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95503358",
   "metadata": {},
   "source": [
    "#### Iterate through each item in the text chunks dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9aeddeea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Create a for loop to iterate through each item in the matching_results_chunks_data dictionary in order to combine all the selected relevant text chunks\n",
    "\n",
    "\"ENTER YOUR CODE HERE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b14e4b",
   "metadata": {},
   "source": [
    "#### Join all the text chunks and store in a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc954cd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Take all of the individual text chunks stored in the context_text list and join them together into a single string named final_context_text. Use \"\\n\" part inserts a newline character between each chunk, effectively creating separate lines or paragraphs.\n",
    "\n",
    "final_context_text = \"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89cba9b5",
   "metadata": {},
   "source": [
    "Proceed to Task 2.5 below (no progress check for Task 2.4 in lab instructions).\n",
    "\n",
    "### Task 2.5. Pass context to Gemini\n",
    "\n",
    "#### Explore the variables of the task\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf8241d2-7b7e-48c8-a7bd-1db5bcdb115f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# \"RUN THIS CELL AS IS\"\n",
    "\n",
    "# You're going to work with provided variables in this task. First, review and describe the content/purpose of each variable below. \n",
    "\n",
    "prompt = f\"\"\" Instructions: Compare the images and the text provided as Context: to answer multiple Question:\n",
    "Make sure to think thoroughly before answering the question and put the necessary steps to arrive at the answer in bullet points for easy explainability.\n",
    "If unsure, respond, \"Not enough context to answer\".\n",
    "\n",
    "Context:\n",
    " - Text Context:\n",
    " {final_context_text}\n",
    "\n",
    "\n",
    "{query}\n",
    "\n",
    "Answer:\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46376c65",
   "metadata": {},
   "source": [
    "#### Generate Gemini response with streaming output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bebe99d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"COMPLETE THE MISSING PART AND RUN THIS CELL\"\n",
    "\n",
    "# Call \"get_gemini_response\" function from utils module in order to generate Gemini response with streaming output. This function uses a multimodal Gemini model, a text prompt, and configuration parameters and instructs the Gemini model to generate a response using the provided prompt. As Gemini model enables streaming, you will receive chunks of the response as they were produced. \n",
    "# Format the streamed output using Markdown syntax for easy readability and conversion to HTML.\n",
    "\n",
    "\"ENTER YOUR CODE HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f691d3-2468-4efd-9040-6b2736ab85df",
   "metadata": {},
   "source": [
    "### To verify your work for Task 2.5, click __Check my progress__ in the lab instructions."
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-cpu.2-15.m119",
   "type": "gcloud",
   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/tf2-cpu.2-15:m119"
  },
  "kernelspec": {
   "display_name": "Python 3 (Local)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
